Evaluating the quality of analytic ratings with Mokken scaling

Greatly influenced by the work of Rasch (1960/1980), Mokken (1971) presented a nonparametric scaling procedure that is based on the theory of invariant measurement, but draws upon less strict requirements related to the scale of measurement. Because they are theoretically and empirically related to Rasch models, Mokken’s nonparametric models have been recognized as a useful exploratory tool for examining data in terms of the basic requirements for invariant measurement before the application of a parametric model. In particular, recent research has explored the use of polytomous versions of Mokken’s (1971) nonparametric scaling models as a technique for evaluating the quality of holistic ratings (Wind & Engelhard, in press) and rating scales (Wind, 2014) for performance assessments in terms of the requirements for invariant measurement. The current study continues the extension of Mokken scaling to performance assessments by exploring the degree to which Mokken-based rating quality indices can be used to explore the quality of ratings assigned within domains on an analytic rubric. Using an illustrative analysis, this study demonstrates the use of a generalized rating design to explore the quality of analytic ratings within the framework of Mokken scaling. Findings from the illustrative analysis suggest that a generalized rating design can be used to examine the quality of analytic ratings in terms of the requirements for invariant measurement.